Software, such as programs or applications, must be tested after each substantial revision to determine if the changes in the new version might have detrimentally affected the operation of the software due to unanticipated conflicts or errors. Software testers utilize a number of testing tools to evaluate the performance of new versions and to identify the source of any problems they may find.
Testing software is a tedious process that must be repeated after each revision. Oftentimes, performance testing starts with a benchmarking test. If the results of the benchmarking test indicate that performance of the software is not as anticipated, then additional software tests are typically performed, this time with one or more testing tools until the source of the problem is identified so that the problem can be corrected.
Each test of the software requires the tester to develop a test scenario in which the tester identifies each testing tool or tools to be used, what data each tool will track, and what operational scenario the software should perform. Typically, for each scenario, a test routine or script is created to specifically test the circumstances surrounding that scenario. There may be multiple test routines for each piece of software code.
During the test, a quality assurance (QA) engineer may discover a bug or bugs and a developer may fix the bug or bugs. As a result, a new version of the software may be generated. Typically, the set of changes to the code between a current version and a previous version of the software, also referred to as a ‘delta’, may be minimal. However, such a change may still require another round of testing to make sure that the new code works properly. In a conventional test framework, the entire set of test routines is run even though the delta is relatively small. As a result, some tests are repeatedly performed even though some code has not been changed, which may consume unnecessary time and resources.